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DEFENSE  MANAGEMENT 

DOD  Needs  to  Demonstrate  That 
Performance-Based  Logistics  Contracts 
Are  Achieving  Expected  Benefits 


Why  GAO  Did  This  Study 


What  GAO  Found 


The  Department  of  Defense  (DOD) 
contracts  with  private  sector 
companies  to  perform  depot 
maintenance  of  weapon  systems 
using  performance-based  logistics 
— that  is,  purchasing  a  defined  level 
of  performance  over  a  defined  time 
period  at  a  fixed  cost  to  the 
government.  After  implementing 
such  contracts,  program  offices  are 
to  validate  their  efficacy  using  cost 
and  performance  data;  DOD  cannot 
otherwise  ensure  cost  savings  and 
improved  performance  are  being 
achieved  through  the  use  of 
performance-based  logistics.  GAO 
was  asked  to  review  the 
implementation  of  performance- 
based  logistics  to  determine 
whether  DOD  could  demonstrate 
cost  savings  and  improved 
responsiveness  from  these 
arrangements.  In  conducting  its 
review,  GAO  analyzed  the 
implementation  of  performance- 
based  logistics  arrangements  for  15 
weapon  system  programs. 


What  GAO  Recommends 


GAO  recommended  that  DOD,  to 
demonstrate  whether  performance- 
based  logistics  contracts  are 
resulting  in  reduced  costs  and 
increased  performance,  develop 
procedures  to  track  whether 
program  offices  validate  their 
business  case  decisions  and  verify 
the  reliability  of  contractor  cost 
and  performance  data.  DOD 
concurred  with  GAO’s 
recommendations. 


DOD  program  offices  could  not  demonstrate  that  they  have  achieved  cost 
savings  or  performance  improvements  through  the  use  of  performance- 
based  logistics  arrangements.  Although  DOD  guidance  on  implementing 
these  arrangements  states  program  offices  should  update  their  business  case 
analysis  based  on  actual  cost  and  performance  data,  only  1  of  the  15 
program  offices  included  in  GAO’s  review  had  performed  such  an  update 
consistent  with  DOD  guidance.  In  the  single  case  where  the  program  office 
had  updated  its  business  case  analysis,  it  determined  that  the  performance- 
based  logistics  contract  did  not  result  in  expected  cost  savings  and  the 
weapon  system  did  not  meet  established  performance  requirements.  In 
general,  program  offices  had  not  updated  their  business  case  analysis  after 
entering  into  a  performance-based  logistics  contract  because  they  assumed 
that  the  costs  for  weapon  system  maintenance  incurred  under  a  fixed-price 
performance-based  logistics  contract  would  always  be  lower  than  costs 
under  a  more  traditional  contracting  approach  and  because  they  lacked 
reliable  cost  and  performance  data  needed  to  validate  assumptions  used. 
Furthermore,  the  Office  of  the  Secretary  of  Defense  has  not  established 
procedures  to  monitor  program  offices  to  ensure  they  follow  guidance  and 
update  the  business  case  analysis.  Additionally,  program  officials  said 
because  of  limitations  in  their  own  information  systems,  they  typically  relied 
on  cost  and  performance  data  generated  by  the  contractors’  information 
systems  to  monitor  performance-based  logistics  contracts.  The  program 
offices,  however,  had  not  determined  whether  contractor-provided  data 
were  sufficiently  reliable  to  update  their  business  case  analysis.  Although 
the  Defense  Contract  Management  Agency  and  the  Defense  Contract  Audit 
Agency  are  most  commonly  used  to  monitor  higher  risk  contracts,  such  as 
cost  plus  contracts,  they  are  potential  resources  available  to  assist  program 
offices  in  monitoring  fixed-price  performance-based  contracts.  In  doing  so, 
these  DOD  agencies  have  the  capability  to  verify  the  reliability  of 
contractors’  information  systems  and  collect  cost  and  performance  data 
needed  to  update  their  business  case  analysis.  Until  program  offices  follow 
DOD’s  guidance  and  update  their  business  case  analysis  based  on  reliable 
cost  and  performance  data,  DOD  cannot  evaluate  the  extent  to  which 
performance-based  logistics  arrangements  are  achieving  expected  benefits 
and  being  effectively  implemented  within  DOD. 


www.gao.gov/cgi-bin/getrpt7GAO-05-966. 

To  view  the  full  product,  including  the  scope 
and  methodology,  click  on  the  link  above. 
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United  States  Government  Accountability  Office 
Washington,  D.C.  20548 


September  9,  2005 

The  Honorable  John  Ensign 
Chairman 

The  Honorable  Daniel  K.  Akaka 
Ranking  Minority  Member 

Subcommittee  on  Readiness  and  Management  Support 
Committee  on  Armed  Services 
United  States  Senate 

The  Department  of  Defense  (DOD)  reported  that  it  spent  $23.6  billion  in 
fiscal  year  2004  for  depot  maintenance  of  military  weapon  systems.1 
Almost  half  of  those  expenditures  were  paid  to  private  sector  companies  to 
perform  depot  maintenance  activities  for  selected  weapon  systems.  DOD’s 
preferred  approach  for  providing  long-term  total  system  support  for 
weapon  systems  is  a  concept  it  calls  performance-based  logistics. 
Performance-based  logistics  is  a  variation  of  other  contractor  logistics 
support  strategies  and  involves  defining  a  level  of  performance  that  the 
weapon  system  is  to  achieve  over  a  period  of  time  at  a  fixed  cost  to  the 
government.2  Specific  performance  measures  are  established  in  each 
contract.  DOD  believes  this  contracting  strategy  for  supporting  weapon 
systems  and  components  will  achieve  higher  levels  of  performance  while 
minimizing  costs.  For  example,  DOD  has  claimed  millions  of  dollars  in  cost 
savings  and  significant  performance  improvements  in  such  areas  as 
material  availability  and  logistics  response  time  through  the  use  of 
performance-based  logistics.  As  a  result  of  aggressively  pursuing  this 
contracting  strategy,  DOD  expected  an  increasing  number  of  weapon 
systems  and  components  to  be  maintained  for  the  military  services  by 
private  sector  companies  under  long-term,  fixed-price,  performance-based 
contracts.  To  justify  the  use  of  a  performance-based  logistics  strategy, 
according  to  DOD  guidance,  program  offices  should  conduct  a  business 
case  analysis  prior  to  adopting  this  approach.  Following  implementation  of 
a  performance-based  logistics  contract,  program  offices  should  update 
their  business  case  analysis  to  validate  the  assumptions  used  based  on 


'Depot  maintenance  encompasses  materiel  maintenance  or  repair  requiring  the  overhauling, 
upgrading,  or  rebuilding  of  parts,  assemblies,  or  subassemblies  and  the  testing  and 
reclamation  of  equipment. 

2Performance-based  logistics  arrangements  for  weapon  system  support  can  include  either  a 
contract  with  a  private  sector  company  or  an  arrangement  to  use  a  government  facility. 
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actual  cost  and  performance  data  and  to  support  future  weapon  systems 
plans. 

As  requested,  we  reviewed  DOD’s  implementation  of  performance-based 
logistics  arrangements.  Our  objective  was  to  determine  whether  DOD 
could  demonstrate  cost  savings  or  improved  responsiveness  through  the 
use  of  performance-based  logistics  arrangements.  This  is  the  second  of 
two  reports  addressing  DOD’s  implementation  of  performance-based 
logistics.  In  an  August  2004  report,3  we  identified  the  types  of  contractor 
logistics  support  arrangements  the  private  sector  used  when  outsourcing 
the  maintenance  of  complex  and  costly  equipment  with  life-cycle 
management  issues  similar  to  those  of  military  systems.  We  also  identified 
several  lessons  learned  from  comparisons  between  private  sector 
contractor  support  practices  and  the  performance-based  logistics  approach 
that  DOD  was  urging  the  services  to  use.  That  report  recommended  that 
DOD  incorporate  into  guidance  the  private  sector’s  practice  of  using 
performance-based  contracts  to  achieve  economies  at  the  component 
level,  rather  than  as  a  preferred  practice  at  the  platform  level.  DOD  issued 
policy  memoranda  and  revised  its  guidebook  to  implement  our 
recommendation. 

As  part  of  our  current  review,  we  collected  and  analyzed  data  on  15  weapon 
system  programs  that  were  among  those  identified  by  the  Office  of  the 
Secretary  of  Defense  and  the  military  services  as  programs  that  have 
successfully  used  performance-based  logistics  arrangements.  We  selected 
weapon  system  programs  that  were  sufficiently  mature  to  provide  lessons 
learned  on  the  use  of  performance-based  logistics.  We  reviewed  contracts 
and  related  files  and  interviewed  program  officials  to  determine  how  these 
arrangements  were  structured  and  managed.  We  examined  business  case 
analysis  documents  prepared  by  the  program  offices  to  justify  the  use  of  a 
performance-based  logistics  strategy.  We  discussed  with  program  officials 
the  systems  they  used  to  monitor  contractor  cost  and  performance.  We 
also  reviewed  DOD  policies,  procedures,  and  guidance  related  to  the  use  of 
performance-based  logistics  for  weapon  system  support.  We  determined 
that  the  data  used  were  sufficiently  reliable  for  our  purposes.  We 
performed  our  work  from  August  2004  through  June  2005  in  accordance 
with  generally  accepted  government  auditing  standards.  The  Scope  and 


3GAO,  Defense  Management:  Opportunities  to  Enhance  the  Implementation  of 
Performance-Based  Logistics,  GAO-04-715  (Washington,  D.C.:  Aug.  16,  2004). 
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Methodology  section  contains  more  detailed  information  about  the  work 
we  performed. 


Results  ill  Brief  DOD  program  offices  could  not  demonstrate  that  they  have  achieved  cost 

savings  or  performance  improvements  through  the  use  of  performance- 
based  logistics  arrangements.  Although  DOD  guidance  on  implementing 
these  arrangements  states  program  offices  should  update  their  business 
case  analysis  based  on  actual  cost  and  performance  data,  only  1  of  the  15 
program  offices  included  in  our  review  had  performed  such  an  update 
consistent  with  DOD  guidance.  In  the  single  case  where  the  program  office 
had  updated  its  business  case  analysis,  it  determined  that  the  performance- 
based  logistics  contract  did  not  result  in  expected  cost  savings  and  the 
weapon  system  did  not  meet  established  performance  requirements.  In 
general,  program  offices  had  not  updated  their  business  case  analysis  after 
entering  into  a  performance-based  logistics  contract  because  they  assumed 
that  the  costs  for  weapon  system  maintenance  incurred  under  a  fixed- 
price,  performance-based  logistics  contract  would  always  be  lower  than 
costs  under  a  more  traditional  contracting  approach  and  because  they 
lacked  reliable  cost  and  performance  data  needed  to  validate  assumptions 
used.  Furthermore,  the  Office  of  the  Secretary  of  Defense  has  not 
established  procedures  to  monitor  program  offices  to  ensure  they  follow 
guidance  and  update  the  business  case  analysis.  Additionally,  program 
officials  said  because  of  limitations  in  their  own  information  systems,  they 
typically  relied  on  cost  and  performance  data  generated  by  the  contractors’ 
information  systems  to  monitor  performance-based  logistics  contracts. 

The  program  offices,  however,  had  not  determined  whether  the  contractor- 
provided  data  were  sufficiently  reliable  to  update  their  business  case 
analysis.  Although  the  Defense  Contract  Management  Agency  (DCMA)  and 
the  Defense  Contract  Audit  Agency  (DCAA)  are  most  commonly  used  to 
monitor  higher  risk  contracts,  such  as  cost  plus  contracts,  they  are 
potential  resources  available  to  assist  program  offices  in  monitoring  fixed- 
price  performance-based  contracts.  In  doing  so,  these  DOD  agencies  have 
the  capability  to  verify  the  reliability  of  contractors’  information  systems 
and  collect  cost  and  performance  data  needed  to  update  the  business  case 
analysis.  Until  program  offices  follow  DOD’s  guidance  and  update  their 
business  case  analysis  based  on  reliable  cost  and  performance  data,  DOD 
cannot  evaluate  the  extent  to  which  performance-based  logistics 
arrangements  are  achieving  expected  benefits  and  being  effectively 
implemented  within  DOD. 


Page  3 


GAO-05 -966  Performance-Based  Logistics 


This  report  contains  recommendations  aimed  at  ensuring  that  DOD 
program  offices  follow  DOD  guidance  for  updating  their  business  case 
analyses  and  verify  the  reliability  of  contractor  cost  and  performance  data. 
In  commenting  on  a  draft  of  this  report,  DOD  concurred  with  our  findings 
and  recommendations.  DOD’s  response  is  included  in  appendix  I. 


Background 


DOD  has  increasingly  relied  on  contractors  to  provide  logistics  support  for 
weapon  system  maintenance.  These  logistics  support  arrangements  have 
taken  various  forms.  In  fiscal  year  1998,  DOD  directed  the  armed  services 
to  pursue  logistics  support  “reengineering”  efforts  with  contractors  to 
achieve  cost  savings  and  improve  efficiency.4  A  1999  DOD  study  identified 
30  pilot  programs  to  test  logistics  support  concepts  that  placed  greater 
reliance  on  the  private  sector.  Some  of  the  pilot  programs  involved 
performance-type  arrangements  that  were  subsequently  converted  to,  or 
designated  as,  performance-based  logistics  contracts.5 6  DOD’s  Quadrennial 
Defense  Review  Report 6  advocated  the  implementation  of  performance- 
based  logistics,  with  appropriate  metrics,  to  compress  the  supply  chain7  by 
removing  steps  in  the  warehousing,  distribution,  and  order  fulfillment 
processes;  reducing  inventories;  and  reducing  overhead  costs  while 
improving  the  readiness  of  major  weapon  systems  and  commodities. 

Over  the  last  few  years,  DOD  has  issued  guidance  on  the  implementation  of 
performance-based  logistics.  In  November  2001,  the  Office  of  the  Deputy 
Under  Secretary  of  Defense  issued  guidance  recommending  that  program 
managers  conduct  a  sound  business  case  analysis  to  decide  whether  they 


4Reengineering  was  the  term  DOD  used  to  describe  its  efforts  to  increase  the  cost- 
effectiveness  of  its  weapon  systems  throughout  their  life  cycles  by  ensuring  that  support 
infrastructures  were  competitive,  timely,  and  efficient. 

5Examples  of  these  logistics  support  arrangements  include  (1)  contractor  logistics  support, 
where  the  contractor  provided  most  or  all  elements  of  logistics  support,  including  depot 
maintenance;  (2)  total  system  performance  responsibility,  under  which  the  contractor 
assumed  responsibility  for  the  weapon  system’s  life-cycle  management;  and  (3)  total  system 
sustainment  responsibility  or  total  system  support  responsibility,  which  gave  the  contractor 
responsibility  for  all  contracted  sustainment  actions  including  parts  management  and  depot- 
level  repair. 

6DOD,  Quadrennial  Defense  Review  Report,  (Washington,  D.C.:  Sept.  30,  2001). 

7Supply  chain  management  refers  to  all  of  the  interrelated  components  and  processes 
required  to  ensure  that  the  correct  amount  of  product  is  in  the  correct  location  at  the  right 
time  and  at  the  lowest  cost. 
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should  implement  performance-based  logistics  for  new  systems  and  major 
acquisitions  for  already  fielded  systems.8  In  an  August  2003  memorandum 
to  the  military  departments,  the  Under  Secretary  of  Defense  (Acquisition, 
Technology  and  Logistics)  stated  that  DOD  should  continue  to  increase  its 
use  of  performance-based  logistics  acquisitions.  On  February  4,  2004,  the 
Deputy  Secretary  of  Defense  (1)  directed  the  Under  Secretary  of  Defense 
(Acquisition,  Technology  and  Logistics),  in  conjunction  with  the  Under 
Secretary  of  Defense  (Comptroller),  to  issue  clear  guidance  on  purchasing 
logistics  support  using  performance  criteria  and  (2)  directed  each  service 
to  provide  a  plan  to  aggressively  implement  performance-based  logistics 
for  current  and  planned  weapon  system  platforms.  Then,  based  on 
recommendations  in  our  August  2004  report,9  the  Under  Secretary  of 
Defense  (Acquisition,  Technology  and  Logistics)  issued  a  memorandum 
reemphasizing  that  the  use  of  this  type  of  support  strategy  was  intended  to 
optimize  weapon  system  availability  while  minimizing  costs  and  the 
logistics  footprint10  and  may  be  applied  to  weapon  systems,  subsystems, 
and  components.  The  memorandum  also  provided  specific  definitions  of 
performance  metrics  to  be  used. 

DOD  describes  performance-based  logistics  as  the  process  of 

(1)  identifying  a  level  of  performance  required  by  the  warfighter  and 

(2)  negotiating  a  performance-based  arrangement  between  the  government 
and  a  contractor  or  government  facility  to  provide  long-term  total  system 
support  for  a  weapon  system  at  a  fixed  level  of  annual  funding.  Instead  of 
buying  spare  parts,  repairs,  tools,  and  data  in  individual  transactions,  DOD 
program  offices  that  use  a  performance-based  logistics  arrangement  buy  a 
predetermined  level  of  performance  that  meets  the  warfighter’s  objectives. 
Although  established  performance  measures  should  be  tailored  to  reflect 
the  unique  circumstances  of  each  performance-based  logistics 
arrangement,  the  measures  are  expected  to  support  five  general  objectives: 
(1)  percentage  of  time  that  a  weapon  system  is  available  for  a  mission 
(operational  availability);  (2)  percentage  of  mission  objectives  met 
(operational  reliability);  (3)  operating  costs  divided  by  a  specified  unit  of 


8DOD,  Product  Support  for  the  21st  Century:  A  Program  Manager’s  Guide  to  Buying 
Performance  (Washington,  D.C.:  Nov.  6,  2001). 

9GAO-04-715. 

“The  logistics  footprint  refers  to  the  size  or  presence  of  the  government  or  contractor 
personnel,  equipment,  inventory,  facilities,  transportation  assets,  or  real  estate  needed  to 
deploy,  sustain,  and  move  the  weapon  systems  or  components. 
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measure  (cost  per  unit  usage);  (4)  size  or  presence  of  support  required  to 
deploy,  sustain,  or  move  a  weapon  system  (logistics  footprint);  and 
(5)  period  of  time  that  is  acceptable  between  the  demand  or  request  for 
support  and  the  satisfactory  fulfillment,  of  that  request  (logistics  response 
time).  Currently,  a  DOD  task  force  is  refining  these  objectives  into  DOD 
standard  performance  definitions  to  be  used  by  program  offices  in  every 
service  when  preparing  performance-based  logistics  arrangements. 

DOD  guidance  recommends  that  program  offices  prepare  a  business  case 
analysis  prior  to  adopting  a  performance-based  logistics  approach  to 
support  a  weapon  system.11  The  aim  of  the  business  case  analysis  is  to 
justify  the  decision  to  enter  into  a  performance-based  logistics  contract. 
The  business  case  analysis  is  to  include  cost  savings  that  are  projected  as  a 
result  of  using  a  performance-based  logistics  approach  and  the 
assumptions  used  in  developing  the  business  case  analysis.  Furthermore, 
DOD  guidance  states  that  program  offices  should  update  their  business 
case  analyses  at  appropriate  decision  points  when  sufficient  cost  and 
performance  data  have  been  collected  to  validate  the  assumptions  used  in 
developing  the  business  case  analyses,  including  the  costs  of  alternative 
approaches,  projected  cost  savings,  and  expected  performance  levels. 
Further,  GAO  Internal  Control  Standards  state  that  it  is  necessary  to 
periodically  review  and  validate  the  propriety  and  integrity  of  program 
performance  measures  and  indicators.  Also,  actual  performance  data 
should  be  continually  compared  against  expected  or  planned  goals,  and 
any  difference  should  be  analyzed.  Additionally,  management  should  have 
a  monitoring  strategy  that  emphasizes  to  program  managers  their 
responsibility  for  internal  controls  (i.e.,  to  review  and  validate  performance 
measures  and  indicators)  and  that  includes  a  plan  for  periodic  evaluation  of 
control  activities. 


uDOD,  Performance-Based  Logistics:  A  Program  Manager’s  Product  Support  Guide 
(Washington,  D.C.:  Sept.  30,  2004);  Product  Support  Strategy  (PSS)  Business  Case 
Analysis  ( BCA )  Guiding  Principles,  (Washington,  D.C.:  Jan.  23,  2004);  and  Business  Case 
Development  Guide  (Washington,  D.C.). 
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DOD  Program  Offices 
Could  Not 
Demonstrate  Cost 
Savings  and 
Performance 
Improvement  Resulting 
from  the  Use  of 
Performance-Based 
Logistics 


DOD  program  offices  could  not  demonstrate  that  their  use  of  performance- 
based  logistics  arrangements  had  achieved  cost  savings  and  performance 
improvements  because  they  had  not  updated  their  business  case  analysis 
as  suggested  by  DOD  guidance.  Specifically,  of  the  15  DOD  program 
offices,  only  1  updated  its  business  case  analysis  to  validate  assumptions 
concerning  cost  and  performance.  Other  DOD  program  offices  had  not 
updated  their  business  case  analysis  in  part  because  they  lacked  reliable 
contractor  cost  and  performance  data.  The  program  offices  typically  relied 
on  cost  and  performance  data  generated  by  contractors’  information 
systems  without  verifying  that  the  data  were  sufficiently  reliable  to  update 
the  business  case  analysis.  Two  DOD  agencies,  DCMA  and  DCAA,  have  the 
capability  to  assist  program  offices  in  monitoring  fixed-price  performance- 
based  contracts,  verifying  the  reliability  of  contractors’  information 
systems,  and  collecting  cost  and  performance  data. 


Projected  Cost  Savings  and 
Performance  Improvements 
Were  Not  Validated 


None  of  the  15  program  offices  included  in  our  review  could  demonstrate 
that  use  of  a  performance-based  logistics  arrangement  had  achieved  cost 
savings  and  performance  improvements.  Although  an  updated  business 
case  analysis  based  on  actual  cost  and  performance  data  might  show  that 
cost  savings  and  performance  improvements  were  being  achieved,  only 
1  of  the  15  program  offices  had  updated  its  business  case  analysis 
consistent  with  DOD  guidance.  Of  the  15  program  offices,  11  had 
developed  a  business  case  analysis  prior  to  entering  into  a  performance- 
based  logistics  arrangement.12  In  their  analysis,  these  program  offices 
projected  that  they  would  achieve  significant  cost  savings.  For  example,  an 


12Four  of  the  15  program  offices  we  reviewed  had  not  prepared  a  business  case  analysis. 
Officials  at  two  program  offices  stated  that  their  contracting  arrangements  did  not  meet  the 
existing  definition  of  a  performance-based  logistics  contract  and,  therefore,  the  requirement 
to  conduct  a  business  case  analysis  was  not  applicable.  Subsequently,  the  Under  Secretary 
of  Defense  (Acquisition,  Technology  and  Logistics)  issued  guidance  on  purchasing  weapon 
systems  logistics  support  using  performance-based  criteria  and  clarified  the  types  of 
contract  arrangements  to  be  considered  as  performance-based  logistics  arrangements. 

DOD  determined  that  under  these  clarified  criteria,  these  two  contracts  meet  the  definition 
of  a  performance-based  logistics  contract.  The  other  two  program  offices  determined  that 
they  lacked  sufficient  cost  and  performance  information  necessary  to  prepare  business  case 
analyses.  To  collect  baseline  cost  and  performance  data,  they  entered  into  cost-plus 
contracts  rather  than  fixed-price  contracts.  On  the  basis  of  an  assessment  of  baseline  data 
collected  in  the  course  of  the  cost-plus  contracts,  the  program  offices  expect  to  determine 
at  a  later  time  if  it  would  be  advantageous  to  transition  from  the  cost-plus  contract  to  fixed- 
price  performance-based  logistics  arrangements.  Before  doing  so,  they  would  prepare  a 
business  case  analysis  as  suggested  under  DOD  guidance. 
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Army  program  office  projected  total  cost  savings  of  $508.5  million,  and  a 
Navy  program  office  projected  cost  savings  of  $29.7  million.  However,  only 
the  Navy’s  T-45  program  office  had  subsequently  updated  its  business  case 
analysis  consistent  with  DOD  guidance  to  determine  whether  cost  savings 
were  being  achieved.13  Realizing  that  the  contractor  was  not  meeting  the 
aircraft  availability  performance  measure,  the  program  office  reassessed 
its  business  case  assumptions  and  found  that  costs  per  flying  hour  were 
higher  than  estimated  because  the  aircraft  was  flying  fewer  hours  than 
forecasted.  As  a  result,  the  program  office  negotiated  separate  contracts 
for  the  airframes  and  engines,  which  resulted  in  estimated  cost  savings  of 
$144  million  over  5  years. 

Performance  indicators  tracked  by  the  program  offices  showed  that  the 
contractors  met  or  exceeded  performance  requirements.  Of  the  15 
programs,  10  reported  that  performance  levels  exceeded  contract 
requirements,  and  5  reported  that  performance  levels  were  meeting 
contract  requirements.  For  example,  an  Army  program  office  reported  a 
weapon  system  availability  rate  of  99  percent,  which  is  7  percent  higher 
than  what  was  projected  in  the  business  case  analysis.  Similarly,  a  Navy 
program  office  reported  a  weapon  system  availability  rate  of  97  percent, 
which  is  7  percent  higher  than  projected. 

Despite  the  reported  performance  improvements,  the  program  offices  had 
not  analyzed  the  performance  data  to  validate  the  improvements  and 
determine  whether  these  improvements  could  be  attributed  directly  to 
their  use  of  performance-based  logistics  arrangements  to  support  the 
weapon  systems.  In  addition,  we  noted  that  program  offices  in  the  past 
reported  they  had  also  met  or  exceeded  required  levels  of  performance 
using  other  contractual  arrangements  for  weapon  system  maintenance. 
Moreover,  the  DOD  program  offices  reporting  that  performance  levels  were 
exceeding  contract  requirements  under  performance-based  logistics 
arrangements  had  not  determined  the  incremental  costs  associated  with 
achieving  these  higher  levels  of  performance.  As  a  result,  they  had  no  way 
of  knowing  whether  incremental  costs  outweighed  the  benefits  derived 
from  achieving  performance  levels  in  excess  of  requirements. 

Program  officials  did  not  follow  DOD  guidance  to  update  and  validate  their 
business  case  analyses  because  they  assumed  that  costs  incurred  under 


13The  T-45  Goshawk  is  a  tandem-seat  jet  trainer  aircraft  used  by  the  Navy  and  Marine  Corps 
for  training  in  carrier  aviation  and  tactical  strike  missions. 
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fixed-price  performance-based  logistics  arrangements  would  always  be 
lower  than  costs  incurred  under  more  traditional  contracting 
arrangements,  and  several  program  officials  cited  a  lack  of  reliable  data 
needed  to  validate  expected  costs  savings  and  improved  performance. 
However,  the  experience  of  the  T-45  program  showed  that  it  is  possible  for 
program  offices  to  validate  the  assumptions  in  the  business  case  analysis 
and  to  determine  whether  expected  cost  savings  and  performance 
improvements  were  achieved.  There  are  also  other  benefits  derived  from 
validating  the  assumptions  used  in  the  business  case  analysis.  Validation 
can  provide  a  better  understanding  of  costs  associated  with  the  repair  and 
maintenance  of  weapon  systems,  ensure  that  proper  performance  metrics 
are  in  place  to  satisfy  logistical  demand,  isolate  incremental  costs 
associated  with  achieving  higher  levels  of  performance,  and  make  cost  and 
performance  data  available  for  contract  renegotiations  in  order  to  obtain 
the  best  value  for  the  government.  Furthermore,  we  did  not  find  evidence 
that  the  Office  of  the  Secretary  of  Defense  had  established  procedures  to 
monitor  whether  program  offices  were  following  its  guidance  to  update 
their  business  case  analyses.  The  results  of  these  updates  could  be  used  by 
DOD  to  assess  the  implementation  of  performance-based  logistics 
arrangements  and  evaluate  the  extent  to  which  performance-based 
logistics  arrangements  are  achieving  expected  benefits. 


DOD  Program  Offices 
Relied  on  Contractors’  Data 
Without  Verifying  the 
Reliability  of  the  Data 


DOD  program  offices  included  in  our  review  stated  that  because  of 
limitations  in  their  own  information  systems,  they  typically  relied  on  cost 
and  performance  data  generated  by  the  contractors’  information  systems  to 
monitor  performance-based  logistics  contracts.  Program  offices 
acknowledged  limitations  in  their  own  information  systems  in  providing 
reliable  data  to  closely  monitor  contractor  cost  and  performance.  Existing 
systems  are  capable  of  collecting  some  cost  and  performance  information 
on  performance-based  logistics  contracts;  however,  according  to  program 
officials,  the  systems  are  not  capturing  sufficiently  detailed  cost  and 
performance  information  for  monitoring  performance-based  logistics 
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contracts.14  Program  officials  told  us  they  had  more  confidence  in  the 
accuracy  and  completeness  of  contractor  systems  than  in  their  legacy 
systems.  The  program  offices,  however,  had  not  determined  whether  the 
contractor-provided  data  were  sufficiently  reliable  to  update  their  business 
case  analyses.  As  a  result,  the  program  offices  did  not  have  the  reliable 
data  they  needed  to  validate  the  assumptions  used  in  the  business  case 
analysis  and  to  determine  whether  their  performance-based  logistics 
arrangements  were  achieving  expected  cost  savings  and  improved 
performance. 

As  we  noted  in  a  prior  report  on  DOD’s  management  of  depot  maintenance 
contracting,16  to  reduce  personnel  and  save  costs,  DOD  decided  to  rely 
more  on  contractors  to  manage  and  oversee  fixed-price  contracts  because 
these  contracts  are  considered  low  risk.  The  contractor  assumes  most  of 
the  risks  for  fixed-priced  contracts,  with  the  government  taking  a  more 
limited  role  in  monitoring  these  contracts.  In  our  prior  work  on  defense 
contract  management,  we  discussed  the  importance  of  monitoring 
contractors’  systems  to  ensure  the  accuracy  and  completeness  of 
information  generated  by  these  systems.16  In  addition,  during  our  review  of 
the  private  sector’s  use  of  performance-based  logistics,17  we  noted  that 
private-sector  companies  that  use  performance-based  logistics  contracts, 
whether  fixed  price  or  cost-plus,  closely  monitor  cost  and  performance 
information  to  effectively  manage  their  contracts.  These  companies  said 
they  rely  on  their  own  systems  and  personnel  to  verify  the  cost  and  quality 
of  work  performed  by  the  contractor.  The  private  sector  takes  this 
approach  (1)  to  ensure  that  expected  costs  under  the  contracts  are 
accurate  and  meet  the  company’s  reliability  standards,  (2)  to  validate  the 
business  case  decision  used  to  justify  a  performance-based  logistics 
arrangement,  and  (3)  to  obtain  the  data  necessary  to  renegotiate  the 
contract. 


14In  a  previous  report,  we  found  that  despite  a  significant  investment  in  business  systems, 
DOD  continues  to  have  long-standing  financial  and  business  management  problems  that 
preclude  the  department  from  producing  reliable  and  timely  information  for  making 
decisions.  See  GAO,  DOD  Business  System  Modernization:  Billions  Being  Invested 
without  Adequate  Oversight,  GAO-05-381  (Washington,  D.C.:  April  29,  2005). 

15GAO,  Defense  Depot  Maintenance:  Contracting  Approaches  Should  Address  Workload 
Characteristics,  GAO/NSIAD-98-130  (Washington,  D.C.:  June  15,  1998). 

16GAO,  High  Risk  Series:  Defense  Contract  Management,  GAO/HR-97-4  (Washington,  D.C.: 
Feb.  1997). 

17See  GAO-04-715. 
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DOD  Program  Offices  Made 
Limited  Use  of  Resources 
Available  to  Assist  in 
Monitoring  Performance- 
Based  Logistics  Contracts 


DCMA  and  DCAA  have  the  capability  to  monitor  contractor  cost  and 
performance,  verify  the  reliability  of  contractor-provided  data,  and  collect 
detailed  cost  and  performance  data.  However,  most  of  the  DOD  program 
offices  we  reviewed  made  limited  use  of  these  agencies’  resources  because 
they  viewed  fixed-price  performance-based  logistics  contracts  to  be  low 
risk  compared  with  other  types  of  contracts.  Before  a  contract  is  awarded, 
DCMA  can  provide  advice  and  service  to  help  construct  effective 
solicitations,  identify  potential  risk,  select  the  most  capable  contractors, 
and  write  contracts  that  meet  the  needs  of  DOD  customers.  After  the 
contract  is  awarded,  DCMA  can  monitor  contractors’  information  systems 
to  ensure  that  cost,  performance,  and  delivery  schedules  are  in  compliance 
with  the  terms  and  conditions  of  the  contracts.  DCAA  performs  contract 
audits  for  DOD  components  and  provides  accounting  and  financial 
advisory  services  during  contract  negotiation  and  administration  of 
contracts. 


DCMA  and  DCAA  officials  said  that  they  have  a  greater  role  in  monitoring 
cost  information  for  cost-plus  contracts  because  such  contracts  are 
considered  high  risk.  According  to  DCMA  and  DCAA  officials,  their  level  of 
oversight  is  significantly  less  for  fixed-priced  contracts,  including 
performance-based  logistics  arrangements,  because  DOD  considers  these 
contracts  to  be  low  risk,  thereby  diminishing  the  need  for  monitoring 
contractor  performance.  Without  a  request  from  program  offices  or 
specific  contract  clauses,  DCMA  and  DCAA  generally  would  not  conduct 
periodic  reviews  or  audits  of  fixed-price  contracts  to  verify  cost  and 
performance  information.  DCMA  and  DCAA  officials  also  said  that  in  the 
past,  monitoring  fixed-price  contracts  was  included  in  their  workload,  but 
because  of  a  reduction  in  staff  and  streamlining  of  operations,  they  focused 
their  efforts  on  contract  areas  that  have  the  highest  risk  for  cost  growth. 
DCMA  and  DCAA  officials  said  they  would  support  increasing  their  role  in 
monitoring  fixed-price  performance-based  contracts  depending  on  the 
availability  of  their  resources. 


Conclusions  DOD  is  expanding  its  use  of  performance-based  logistics  as  its  preferred 

support  strategy  in  support  of  weapon  systems  but  has  not  yet 
demonstrated  that  this  long-term  support  strategy  is  being  effectively 
implemented  DOD-wide.  DOD  guidance  states  that  program  offices,  after 
entering  into  performance-based  logistics  arrangements,  should  update 
their  original  business  case  analysis  using  actual  cost  and  performance 
data  to  validate  their  assumptions,  but  most  of  the  program  offices  we 
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reviewed  had  not  followed  this  guidance,  and  the  Office  of  the  Secretary  of 
Defense  was  not  monitoring  whether  program  offices  were  following  the 
guidance.  The  program  offices  therefore  could  not  substantiate  that  cost 
savings  and  performance  improvements  for  weapon  system  support  were 
being  achieved  through  the  use  of  performance-based  logistics 
arrangements.  Program  offices  also  have  lacked  reliable  cost  and 
performance  data  needed  to  validate  the  results  of  performance-based 
logistics  arrangements.  Reliable  data  could  be  collected  and  analyzed  by 
increasing  oversight  of  these  contracts  with  the  assistance  of  DCMA  and 
DCAA. 


Recommendations 
Executive  Action 


for 


To  demonstrate  that  performance-based  logistics  arrangements  are 
resulting  in  reduced  costs  and  increased  performance,  and  to  improve 
oversight  of  performance-based  logistics  contracts,  we  recommend  that 
the  Secretary  of  Defense  direct  the  Under  Secretary  of  Defense 
(Acquisition,  Technology  and  Logistics)  to  take  the  following  two  actions: 


1.  Reaffirm  DOD  guidance  that  program  offices  update  their  business 
case  analyses  following  implementation  of  a  performance-based 
logistics  arrangement  and  develop  procedures,  in  conjunction  with  the 
military  services,  to  track  whether  program  offices  that  enter  into  these 
arrangements  validate  their  business  case  decisions  consistent  with 
DOD  guidance. 

2.  Direct  program  offices  to  improve  their  monitoring  of  performance- 
based  logistics  arrangements  by  verifying  the  reliability  of  contractor 
cost  and  performance  data.  The  program  offices  may  wish  to  increase 
the  role  of  DCMA  and  DCAA  in  overseeing  performance-based  logistics 
contracts. 


Agency  Comments  and 
Our  Evaluation 


In  commenting  on  a  draft  of  this  report,  DOD  concurred  with  our 
recommendations  regarding  the  validation  of  business  case  decisions  for 
performance-based  logistics  arrangements  and  verification  of  reliability  of 
contractor  data.  While  DOD  was  generally  responsive  to  our 
recommendations,  specific  details  on  how  DOD  planned  to  validate  and 
verify  contractor  data  were  not  provided. 


Regarding  our  recommendation  to  reaffirm  guidance  and  develop 
procedures  to  track  whether  program  offices  validate  their  business  case 
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decisions,  DOD  stated  that  the  department  will  reaffirm  DOD  guidance  on 
updating  the  business  case  analysis  after  implementing  performance-based 
logistics  arrangements  and  will  work  with  the  military  services  to  develop 
procedures  to  track  whether  program  offices  validate  their  business  case 
decisions  consistent  with  DOD  guidance. 

With  regard  to  our  second  recommendation  to  direct  program  offices  to 
verify  the  reliability  of  contractor  cost  and  performance  data,  DOD  stated 
that  it  will  issue  guidance  on  verifying  the  reliability  of  contractor  cost  and 
performance  data.  DOD  did  not  provide  specific  information  on  what  the 
guidance  would  include  nor  did  it  indicate  whether  it  would  increase  the 
use  of  DCMA  or  DCAA  to  verify  the  reliability  of  contractor  cost  and 
performance  data. 

DOD  also  provided  technical  comments,  which  we  have  incorporated  as 
appropriate. 


Scope  and 
Methodology 


To  determine  whether  DOD  could  demonstrate  that  cost  savings  and 
improved  performance  were  being  achieved  through  the  use  of 
performance-based  logistics  arrangements,  we  collected  and  analyzed  data 
on  15  weapon  system  programs  identified  by  the  Office  of  the  Secretary  of 
Defense  and  the  military  services  as  programs  that  have  successfully  used 
performance-based  logistics  arrangements.  The  15  programs  are  fisted  in 
table  1. 


Table  1:  Performance-Based  Logistics  Programs  Included  in  Our  Review 

Military  service 

Weapon  system  or 
component 

Program  office 

Air  Force 

C-17 

Air  Force  Materiel  Command 

Air  Force 

F-117 

Air  Force  Materiel  Command 

Air  Force 

JSTARS 

Warner  Robins  Air  Logistics  Center 

Air  Force 

C-130J 

Air  Force  Materiel  Command 

Navy 

ALR-67  (V3) 

Naval  Supply  Systems  Command 

Navy 

Auxiliary  Power  Units 

Naval  Supply  Systems  Command  and  Naval  Air  Systems  Command 

Navy 

F-18  E/F  FIRST 

Naval  Supply  Systems  Command  and  Naval  Air  Systems  Command 

Navy 

F-404 

Naval  Supply  Systems  Command  and  Naval  Air  Systems  Command 

Navy 

T-45  engines 

Naval  Supply  Systems  Command  and  Naval  Air  Systems  Command 
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(Continued  From  Previous  Page) 

Military  service 

Weapon  system  or 
component 

Program  office 

Navy 

V- 22  engines  a 

Naval  Air  Systems  Command 

Navy/Marine  Corps 

KC-130J  b 

Naval  Supply  Systems  Command  and  Naval  Air  Systems  Command 

Army 

HIMARS 

Aviation  and  Missile  Command 

Army 

Javelin  CLU 

Aviation  and  Missile  Command 

Army 

TOW-ITAS 

Aviation  and  Missile  Command 

Army 

TUAV  Shadow 

Aviation  and  Missile  Command 

Source:  GAO  analysis  of  services’  data. 

a  This  is  a  joint  contract  with  the  Navy  and  Marine  Corps. 
b  The  Marine  Corps’  maintenance  contracting  was  handled  by  a  Navy  team. 


We  reviewed  DOD  and  service  policies,  procedures,  and  guidance  related 
to  the  use  of  performance-based  logistics  and  met  with  program  officials  to 
discuss  how  their  performance-based  logistics  contracts  were  structured 
and  managed  and  how  these  contracts  were  validated  to  ensure  that  cost 
savings  and  improved  performance  were  being  achieved  as  a  result  of  using 
performance-based  logistics.  We  also  obtained  and  analyzed  available 
documentation,  including  business  case  analyses,  contracts,  and  related 
files.  We  did  not  assess  the  methodology  program  offices  used  to  prepare 
their  business  case  analyses  or  the  quality  of  these  analyses.  We  discussed 
with  program  officials  the  systems  they  used  to  monitor  contractor  cost 
and  performance.  We  also  interviewed  officials  from  the  Office  of  the 
Secretary  of  Defense  and  military  department  headquarters  to  discuss 
implementation  of  performance-based  logistics,  lessons  learned,  and  the 
benefits  derived  from  using  performance-based  logistics  approaches  and 
practices.  To  determine  how  private-sector  companies  ensure  that  cost 
and  performance  levels  under  a  performance-based  contract  are  as 
expected,  we  reviewed  the  information  provided  by  seven  companies 
identified  in  our  prior  report  that  used  complex  and  costly  equipment  that 
had  life-cycle  management  issues  similar  to  military  weapon  systems,  and 
outsourced  some  portion  of  their  maintenance  work  under  performance- 
based  contracts.  These  seven  companies  consisted  of  six  airline 
companies  and  one  mining  company.  We  contacted  officials  at  DCMA  and 
DCAA  to  determine  those  agencies’  roles  in  monitoring  the  costs  and 
performance  of  fixed-priced  contracts,  including  performance-based 
logistics  contracts,  how  audits  are  requested  or  initiated,  and  the 
procedures  for  reporting  the  results  of  the  audits. 
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We  are  sending  this  report  to  the  Chairman  and  Ranking  Minority  Member, 
Senate  Subcommittee  on  Readiness  and  Management  Support,  Committee 
on  Armed  Services.  We  will  also  send  copies  to  the  Under  Secretary  of 
Defense  (Acquisition,  Technology  and  Logistics).  Copies  of  this  report  will 
be  made  available  to  others  upon  request.  In  addition,  the  report  will  be 
available  at  no  charge  on  GAO’s  Web  site  at  http://www.gao.gov.  If  you  or 
your  staff  have  any  questions  on  the  matters  discussed  in  this  report,  please 
contact  me  at  (202)  512-8412  or  sohsw@gao.gov.  Contact  points  for  our 
Offices  of  Congressional  Relations  and  Public  Affairs  may  be  found  on  the 
last  page  of  this  report.  GAO  staff  who  made  major  contributions  to  this 
report  are  listed  in  appendix  II. 


William  M.  Solis 

Director,  Defense  Capabilities  and  Management 
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Appendix  I 

Comments  from  the  Department  of  Defense 


6*§ 


Deputy  Under  Secretary  of  Defense  for 
Logistics  and  Materiel  Readiness 

3500  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-3500 


August  30,  2005 


Mr.  William  M.  Solis 

Director,  Defense  Capabilities  and  Management 
U.S.  Government  Accountability  Office 
441  G  Street,  N.W. 

Washington,  DC  20548 

Dear  Mr.  Solis: 


This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  draft  report,  “DEFENSE 
MANAGEMENT:  DoD  Needs  to  Demonstrate  That  Performance-Based  Logistics  Contracts  Are 
Achieving  Expected  Benefits,”  dated  August  16,  2005,  (GAO  Code  350551  /GAO-05-966). 

The  Department  generally  concurs  with  the  report  and  concurs  with  the 
recommendations.  Detailed  comments  on  the  GAO  recommendations  are  provided  in  the 
attachment.  The  Department  appreciates  the  opportunity  to  comment  on  the  draft  report. 

Sincerely, 

Jack  Bell 
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Appendix  I 

Comments  from  the  Department  of  Defense 


GAO  DRAFT  REPORT  -  DATED  August  16,  2005 
GAO  CODE  35055 1 / GAO-05-966 

"  DEFENSE  MANAGEMENT:  DOD  Needs  to  Demonstrate  That  Performance-Based 
Logistics  Contracts  Are  Achieving  Expected  Benefits  " 


DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  RECOMMENDATIONS 


RECOMMENDATION  1:  The  GAO  recommended  that  the  Secretary  of  Defense  direct  the 
Under  Secretary  of  Defense  (Acquisition,  Technology  and  Logistics)  to  reaffirm  DoD  guidance 
that  program  offices  update  their  business  case  analysis  following  implementation  of  a 
performance-based  logistics  arrangement  and  develop  procedures,  in  conjunction  with  the 
military  services,  to  track  whether  program  offices  that  enter  into  these  arrangements  validate 
their  business  case  decisions  consistent  with  DoD  guidance.  (Pages  9  and  10/GAO  Draft 
Report) 

DOD  RESPONSE:  Concur.  The  Office  of  the  Under  Secretary  of  Defense  (Acquisition, 
Technology  and  Logistics)  will  reaffirm  DoD  guidance  relating  to  updating  of  the  business  case 
analysis  following  implementation  of  a  performance  based  logistics  arrangement  as  well  as 
work  with  the  military  departments  to  develop  an  appropriate  method  to  track  whether  program 
offices  validate  their  business  case  decisions  consistent  with  DoD  guidance. 

RECOMMENDATION  2:  The  GAO  recommended  that  the  Secretary  of  Defense  direct  the 
Under  Secretary  of  Defense  (Acquisition,  Technology  and  Logistics)  to  direct  program  offices 
to  improve  their  monitoring  of  performance-based  logistics  arrangements  by  verifying  the 
reliability  of  contractor  cost  and  performance  data.  The  program  offices  may  wish  to  increase 
the  role  of  DCMA  and  DCAA  in  overseeing  performance-based  logistics  contracts. 

(Page  10/GAO  Draft  Report) 

DOD  RESPONSE:  Concur.  The  Office  of  the  Under  Secretary  of  Defense  (Acquisition, 
Technology  and  Logistics)  will  issue  guidance  on  verifying  the  reliability  of  contractor  cost  and 
performance  data. 
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Appendix  II 


GAO  Contact  and  Staff  Acknowledgments 


GAO  Contact  William  M.  Solis  (202)  512-8412  or  solisw@gao.gov. 


Acknowledgments 


In  addition  to  the  contact  named  above,  Thomas  Gosling,  Assistant 
Director;  Thom  Barger;  Judith  Collins;  Pamela  Valentine;  and  Cheryl 
Weissman  were  major  contributors  to  this  report. 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and  policies; 
and  provides  analyses,  recommendations,  and  other  assistance  to  help 
Congress  make  informed  oversight,  policy,  and  funding  decisions.  GAO’s 
commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost 
is  through  GAO’s  Web  site  (www.gao.gov).  Each  weekday,  GAO  posts 
newly  released  reports,  testimony,  and  correspondence  on  its  Web  site.  To 
have  GAO  e-mail  you  a  list  of  newly  posted  products  every  afternoon,  go  to 
www.gao.gov  and  select  “Subscribe  to  Updates.” 

Order  by  Mail  or  Phone 

The  first  copy  of  each  printed  report  is  free.  Additional  copies  are  $2  each. 
A  check  or  money  order  should  be  made  out  to  the  Superintendent  of 
Documents.  GAO  also  accepts  VISA  and  Mastercard.  Orders  for  100  or 
more  copies  mailed  to  a  single  address  are  discounted  25  percent.  Orders 
should  be  sent  to: 

U.S.  Government  Accountability  Office 

441  G  Street  NW,  Room  LM 

Washington,  D.C.  20548 

To  order  by  Phone:  Voice:  (202)  512-6000 

TDD:  (202)512-2537 

Fax:  (202)512-6061 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Web  site:  www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 

Relations 

Gloria  Jarmon,  Managing  Director,  JarmonG@gao.gov  (202)  512-4400 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7125 
Washington,  D.C.  20548 

Public  Affairs 

Paul  Anderson,  Managing  Director,  AndersonPl@gao.gov  (202)  512-4800 
U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7149 
Washington,  D.C.  20548 
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